Maria writes:
A) Your presentation Test Strategy: “What is it? What does it look like?” applies to creating a test strategy for a specific application (I’ve also read Ingrid B. Ottevanger’s article on “A Risk-Based Test Strategy”). How can I apply the idea to an overall test strategy for the company that I’m working for? Is it possible to create a really good test strategy for a company so that it covers several diverse applications? Having difficulties in finding a way to create a non-poorly stated strategy from which we can create an efficient test process it leaves me with another question: “How do we make a clear line between the overall test strategy and the company test process?”
B) The precondition for this activity is unfortunately not the best leaving us with a tight time schedule and very little time to do a thorough work. My concern is that neither the strategy nor the test process will actually be something possible to use, leaving us with as you say “A string of test technique buzzwords.” So how can I argue that the test strategy and the test process are not just two documents that we have to create but it’s the thoughts behind the documents that are important.
Test strategy is an important yet little-described aspect of test methodology. Let me introduce three definitions:
Test Plan: the set of ideas that guide a test project
Test Strategy: the set of ideas that guide test design
Test Logistics: the set of ideas that guide the application of resources to fulfill a test strategy
I find these ideas to be a useful jumping off point. Here are some implications:
- The test plan is the sum of test strategy and test logistics.
- The test plan document does not necessarily contain a test plan. This is because many test plan documents are created by people who are following templates without understanding them, or writing things to please their bosses, without knowing how to fulfill their promises, or simply because it once was a genuine test plan but now is obsolete.
- Conversely, a genuine test plan is not necessarily documented. This is because new ideas may occur to you each day that change how you test. In my career, I have mostly operated without written test plans.
One quick way to think about test strategy is to realize that testing is (usually) a process of constructing an explanation of the status of the product. Therefore, the ideas that should guide our testing are those that relate to the marshalling of evidence for that explanation.
Here’s an example: Let’s say I need to test Inkscape, an open source painting program. The people I work for want this product to be a viable alternative to Adobe Photoshop.
This leads to an overarching question for the testing: “Given the general capabilities of Inkscape, is the program sufficiently reliable and are its capabilities well enough deployed that a serious user would consider that Inkscape is a viable alternative to Photoshop?” This is a question about the status of Inkscape. Answering it is not merely a processing of determining yes or no, because, as a tester, I must supply an explanation that justifies my answer.
Working backwards, I would have to do something like the following:
- Catalog the capabilities of Inkscape and match them to Photoshop.
- Determine the major objectives users might have in using a paint program, as well as various kinds of users.
- Learn about the product. Get a feel for it in terms of its structures, functions, data, and platforms.
- List the general kinds of problems that might occur in the product, based on my knowledge of the technology and the users.
- Decide which parts of the product are more likely to fail and/or are more important. Focus more attention on those areas.
- Determine what kinds of operations I need to do and which systematic observations I need to make in order to detect problems in the product (area by area and capability by capability) and compare it to Photoshop. (Here’s where I would also apply a variety of test techniques.)
- Carry out those test activities and repeat as necessary.
- Consider testability and automation as I go.
In doing these things, I would be gathering the evidence I need to argue for the specific ways in which Inkscape does or does not stand up to Photoshop.
Company-wide Test Strategy
In my way of thinking, a good test strategy is product specific. You can have a generic test strategy, but since you don’t test generic products, but only specific products, it will become better when it is made specific to what you are testing at the moment.
Perhaps what you are talking about is a strategy that relates to what you need for a test lab infrastructure, or for developing the appropriate product-specific test skills? Or perhaps you are thinking of creating materials to aid test leads in producing specific test strategies?
If so, one thing to consider is a risk catalog (aka bug taxonomy). A risk catalog is an outline of the kinds of problems that typically occur in products that use a particular technology. You can create one of these based on your experience testing a product, then reuse it for any other similar product.
Company Test Process
I suggest using the term methodology instead of process. “Process” is the way things happen. “Methodology” is a system of methods. You use a methodology; but you participate in a process. When you have an idea and say, “I think I’ll try doing that” the idea itself is probably a method. When you do it, it becomes a practice (a practice is something that you actually do), and therefore it influences the process.
I use these words carefully because process is a very rich and complex reality that I do not want to oversimplify. For instance, it may be a part of your methodology to create a test plan, but at the same time it may be a genuine part of your process that your test plan document is ignored. “Ignore the test plan document” is not going to be written down in anyone’s methodology, yet it can be an important part of the process, especially if the test plan document is full of bad ideas.
The dividing line between test strategy and test methodology is not hard to find, I think. A test strategy is product specific, and a test methodology is not. Another important element you haven’t mentioned is test skill. Your methodology is useless without skilled testers to apply it.
I would suggest that a more important dividing line for you to consider is the line between skill and method. How much do you rely on skilled people to select the right thing to do next, and how much are you trying to program them with methodology? Many process people get this all mixed up. They treat testers as if they are children, or idiots, trying to dictate solutions to problems instead of letting the testers solve the problems for themselves. What the process people then get is either bad testing, or, hopefully, their methodology is ignored and the testers do a good job anyway.
When I develop a test methodology, as I have done for a number of companies, I focus on training the testers and on creating a systematic training and coaching mechanism. That way the methodology documentation is much thinner and less expensive to maintain.
James,
You say it best with:
“I would suggest that a more important dividing line for you to consider is the line between skill and method. How much do you rely on skilled people to select the right thing to do next…”
This is one of the biggest reasons that I am changing my company instead of changing my company. The development team here is placing too much emphasis on a development plan instead of improving the teams development skills or changing the team all together. Emphasis placed on plans restricts talented developers, and in this case, testers, and limits their creative abilities. IMO, without creativity, we lose.
I created very detailed test plans for my latest project that were invaluable.
[James’ Reply: So have I. But how do you know they were invaluable?]
I didn’t know the difference between test plan or strategy.
[James’ Reply: If you read my post, now you know.]
I just created a document that listed out elements that needed to be considered for testing. I referred to it often to keep me on track.
[James’ Reply: That is not a detailed test plan, if that’s all you wrote. That sounds like what I call a product coverage outline.]
As far as process people… that is me. I have been on projects that were highly processed that worked like clockwork or a well oiled machine… delivered on time and on budget.
[James’ Reply: Are humans like clocks, then? If so, can you tell me exactly by what mechanism humans are predictable? What is the nature of this machine that you have oiled? Who designed this machine? By what principles did they design it and what part did they play in it daily operation? These are important questions. What Context-Driven people do is to get into the design of things, rather than taking the black box of “process” for granted.
I’m not saying that you didn’t experience something like a well-oiled machine. I’m saying that the oiliness of your machine is not relevant. What is relevant is the effectiveness of that machine. As someone who creates these machines, myself, I know that is not a trivial issue. So, I’m curious why you have chosen to say nothing about it, and instead emphasize how smoothly your mysterious process performs.]
i have been on projects where the philosophy was to allow creativity and they were nothing but chaos…
[James’ Reply: I wonder if you understand what you are saying, and if you mean what I think you mean? Do you understand that creativity in the technical world means “skilled problem-solving?” Are you criticizing skilled problem-solving as a concept? Are you suggesting that chaos always results when people are allowed to use their minds to solve problems? OR, are you referring to a badly managed, badly executed project, where the people involved didn’t know how to do their jobs?]
I think this idea is used by people that don’t like discipline.
[James’ Reply: I definitely am suspicious of discipline, because that concept is so often used as an excuse to ignore important problems. Discipline is marvelously simplifying isn’t it? “Just shut up and follow the plan.” Well, no, I won’t do that, because that would be irresponsible.
Although I’m suspicious of discipline, I do actually have a certain kind of discipline– which is the discipline to study my craft intensively, wherever that takes me. I have the discipline to leave ignorance behind, even when it’s comfortable and familiar. And I have the discipline to read a blog post and respond to it with a coherent argument of some kind, rather than a childish and insulting accusation. I would like you to have that discipline, too.]
Expand that idea to building a house and see how it turns out. I have never seen a project success with this philosophy. You can still be creative with in process.
[James’ Reply: You are misusing the word “process.” Please go see a dictionary. Process is how things happen. We all have a process. What you seem to be arguing for is a particular kind of process which involves people writing instructions down in detail and then following those instructions. I think when you say process you mean “procedure.” Look that up, too. You say that you have seen a process guided by lots of detailed procedures work and you have not seen alternatives work. Well, I have seen it ALL work and I have seen it ALL fail. I understand that success and failure are contingent on many factors and that I (and you) must develop insight into what those factors are.
Your comment is written on the level of a fairy tale. Something like Hansel and Gretel. Really, you can do better than that.]
hi James, when I read your stuff, I really think, here’s a guy with practical experience, who knows how things really work (or don’t). I’m afraid my boss has given me a test plan template, with the usual nonsense in it. And now, rather than doing something useful, I’ll have to fill out this template with all sorts of nonsense.
[James’ Reply: Did he give it to you because the way you were doing test documentation wasn’t pleasing to him? What happened when you spoke to him about that?]
Some examples from the template I’ve got: a ‘suspension criterion’ is that the test environment is too unstable. So we need a document to tell us that we can’t test if the test environment is totally unstable??
[James’ Reply: Just ignore that part.]
If we need to document things like that, then we’re really in trouble, because we need to be told the blindingly obvious. It’s also got the usual section entitled ‘Risks’, so I’ll be forced to spend time dreaming up some ‘risks’ and mitigations. On some projects, sure, it may be worthwhile to think carefully about potential risks and mitigations. But my project is relatively small, simple and low risk. Yet, of course, because the template has a ‘Risks’ section, I’ll have to dream up some risks, or the boss will say, ‘what, you don’t have any risks? You’ve GOT to have risks, the template says so.’
[James’ Reply: Ask him if he really cares about that. Ask him why he cares. How will documenting that make a difference?]
So again, I can probably say a risk is that we won’t be able to test adequately in the time allowed, or that we might find a defect that we can’t fix in time for go-live. But again, these things are self-evident, and apply to ANY project. So again, I am forced to waste time documenting the blindingly obvious. Sorry, I felt the need to vent.
[James’ Reply: Don’t document project risks. Document product risks. Basically, list the five kinds of bugs you are expecting to see.]
James,
I struggle with the sentence: “What the process people then get is either bad testing, or, hopefully, their methodology is ignored and the testers do a good job anyway.”
[James’ Reply: Thank you. I struggle with that, too.]
This is hard for me because I think it’s harmful if testers ignore dictated bad methodology and just do good work instead.
[James’ Reply: It may be harmful in some senses, and helpful in other senses.]
Obviously it doesn’t help anyone to do bad work. Most managers will actually be disappointed with the results, and the testers will be miserable. So I would not advocate blindly following bad methodology just because you were told to.
But just ignoring their methodology doesn’t seem to be helpful either. If the people pushing the bad methodology don’t know that they were ignored, they will assume that their methods were what ultimately caused the success of the testing, and not the skilled testers who ignored them. This could encourage them to continue to insist on the use of bad ideas.
[James’ Reply: I get it. So, it’s down to a personal ethical decision that may relate to various strategies for making the world a better place. For instance, my disobedience may be a delaying action to buy me time to convince my employer to work in a better way. Or I may be practicing to make myself more skilled so that I can serve the company and society better in the long run. Or I may be following a higher ideal related to doing no harm or honoring my employment contract rather than the stated desires of a crazy employee I happen to work for.]
Knowing that you do not shy away from confrontation, your statement probably implies that as part of ignoring the methodology the tester should explain why they aren’t using it.
[James’ Reply: That what I do, now. I’ve arranged my life to avoid the problem you raise. No one pays me for secret work, anymore. It’s either out in the open or they don’t pay. But when I worked for “the man” I was knee deep in this dilemma.]
But, I think that many people (most?) are less confrontational. I think without explicitly stating the idea of formally opposing the bad methodology, many will just assume doing good work quietly is the best solution.
[James’ Reply: One does not follow from the other. You can be terrified of conflict and yet not *assume* that constructive insubordination is a good idea. I think what you maybe meant to point out is that (in the statement from my post that you cited) *I* assumed it! I shouldn’t have been so flip about that. It’s a choice I have made many times, but each time it is an ethical quandary.]
As most things in our industry, I think a big part of the answer is “it depends”. There are many factors that will always influence decisions. For example, making stands against bad ideas that come down from above you can jeopardize your career (with that employer).
[James’ Reply: Of course.]
You speak often about working with integrity; I believe that includes not only ensuring we do the best work we know how, but ensure the people we’re doing that work for don’t misunderstand what it is we’re doing.
[James’ Reply: Perhaps “ensure” is too strong a word, but I get what you are saying. I also understand that it could be reasonable to silently do good work against even direct orders by benignly incompetent people. Of course, that entails risk. Welcome to the human condition.
Thank you for pointing out that this is a matter to be thought through carefully. You’re right.]
Thanks for the wealth of ideas you share.
Dan